Histological image analysis

ABSTRACT

A machine learning algorithm is trained on a number of microscopic images and a measure of outcome of each image. Each image is divided into tiles. The measure of outcome is assigned to each tile of the image. The tiles are then used to train the machine learning algorithm. The trained algorithm may then be used to evaluate images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/763,860, filed on May 13, 2020, which is a National Stage Applicationof International Application No. PCT/EP2018/080828, filed on Nov. 9,2018, which claims benefit of Application No. 1718970.5, filed on Nov.16, 2017 in Great Britain (GB) and which applications are incorporatedherein by reference. A claim of priority to all, to the extentappropriate, is made.

FIELD OF THE INVENTION

This invention relates to analysis of histological images. It relates inparticular to using a machine-learning algorithm to perform suchanalysis and also to training the machine-learning algorithm to performthe analysis.

BACKGROUND OF THE INVENTION

As used herein, a “histological image” refers to an image showing themicroscopic structure of organic tissue. A “histological feature ofinterest” means a feature of this microscopic structure. The feature maybe of interest for diagnostic or therapeutic purposes, or for scientificresearch, for instance.

Histological specimens are typically used to review the structure todetermine the diagnosis or try to determine a prognosis.

In the case where the histological images relate to pathologies, theterm “histopathological image” may be used.

At the microscopic scale, many of the interesting features of cells arenot naturally visible, because they are transparent and colourless. Toreveal these features, specimens are commonly stained with a markerbefore being imaged under a microscope. The marker includes one or morecolorants (dyes or pigments) that are designed to bind specifically toparticular components of the cell structure, thus revealing thehistological feature of interest.

One commonly used staining system is called H&E (Haematoxylin andEosin). H&E contains the two dyes haematoxylin and eosin. Eosin is anacidic dye—it is negatively charged. It stains basic (or acidophilic)structures red or pink. Haematoxylin can be considered as a basic dye.It is used to stain acidic (or basophilic) structures a purplish blue.

DNA (heterochromatin and the nucleolus) in the nucleus, and RNA inribosomes and in the rough endoplasmic reticulum are both acidic, and sohaematoxylin binds to them and stains them purple. Some extracellularmaterials (i.e. carbohydrates in cartilage) are also basophilic. Mostproteins in the cytoplasm are basic, and so eosin binds to theseproteins and stains them pink. This includes cytoplasmic filaments inmuscle cells, intracellular membranes, and extracellular fibres.

Those skilled in the art will be aware of a number of alternative stainsthat may be used. Such histological images may be used in particular forevaluating tissues that may be cancerous. It is useful to be able toclassify images to determine the expected outcome.

Conventionally, histological features of interest are identified inhistological images by histopathologists—specialist medical expertstrained in the interpretation of these images.

However, experiments have been carried out and the classification byhistopathologists has been shown to be inconsistent and in many cases oflimited prognostic value, both when comparing the identifications ofdifferent histopathologists and even when presenting the same images tothe same histopathologist on different occasions. Such inconsistenciesand inter and intra observer variability can have serious implications.

There is accordingly a need for an automated image analysis approach forcarrying out this task.

One prior art approach to automated image analysis is taught by U.S.Pat. No. 8,467,590. This proposes the use of a variety of pre-determinedstatistical tests and analyses applied to an image and combines themeasures to produce an output. However, it is difficult to designsuitable automated image analysis methods and apparatus as there is awide variety of potentially useful statistical tests and analyses thatmight in principle be used for analysing histological images andextensive research work and testing is often needed to identify the bestapproaches.

There is accordingly a need for improved automated histopathologicalimage analysis methods and apparatus.

SUMMARY OF THE INVENTION

The invention is defined by the claims.

According to a first aspect of the invention, there is provided acomputer-implemented method of training a machine-learning algorithm toperform histopathological image analysis, the method comprising:

obtaining, by the computer, a plurality of microscopic images ofhistological specimens and a measure of outcome for each of the images;dividing each of the plurality of microscopic images into a plurality oftiles; and training a machine-learning algorithm to generate a scorerepresentative of the outcome using as training inputs the plurality oftiles and the outcome for the respective image from which each tile isdivided.

By using a machine-learning algorithm the need for research work toidentify suitable image analysis methods for histological images isavoided. Surprisingly, it is possible to obtain a useful trainedmachine-learning algorithm by dividing each of the plurality ofmicroscopic images into tiles and ascribing the same measure of outcometo each of the tiles of the image.

Such a trained algorithm may deliver significant

The number of tiles may be relatively large—each microscopic image maybe divided into at least 250 tiles, further preferably at least 500tiles. Each of these tiles is deemed for training purposes to share thesame outcome. Thus, a relatively limited number of microscopic imagesmay generate a large number of tiles that may be used for training thealgorithm.

It will be appreciated that the tiles of a single image will vary andeven in the case of a specimen which results in a poor outcome many ofthe tiles may well show no evidence of this. For example, where themicroscopic images are of potentially cancerous tissue sections, anycancer present will only be present in some regions of the microscopicimages and hence only some tiles will include images of such tissue. Theremainder of the tiles will not contribute to the outcome of thecase—indeed the remainder of the tiles may show normal tissue. Therewill therefore be many tiles where there is no indication of anyparticular relevant pathology for the classification of the case.Nevertheless, the machine learning algorithm is trained using the sameoutcome associated with all tiles of a microscopic image. This meansthat the machine-learning algorithm will be trained with many tilesshowing no obvious relevant structure even when there is a poor outcome.There is therefore no clear one-to-one correspondence between thecontents of a tile and the outcome.

At first sight, therefore, it is not clear that a machine-learningalgorithm can be trained on individual tiles being parts of histologicalimages to produce any kind of useful output.

However, the inventors have carried out experiments and even in spite ofthis issue it turns out that reliable and useful results can beobtained. In general, machine learning algorithms trained to provide anoutput for each tile will provide an output for a test image for thetiles of that image, which is not what is required. Normally, to train amachine learning algorithm to provide an output for an image, themachine learning algorithms will be trained using training datacomprising the images and the outcome corresponding to the image.However, this may cause difficulties in view of the need for very largenumbers of images and the very large size of each of these images whichcan cause problems for machine learning algorithms and availablehardware limitation since the algorithm will need to determine itselfwhich of the very large number of features represented by the pixels ofthe image are useful. The inventors have realised that it is possible toprovide a useful training to output data for a complete image even ifthe machine-learning algorithm is trained on individual tiles.

In some embodiments, a single score is obtained for a complete image bythresholding. To explain further, the single predicted outcome value foran image may be achieved by double thresholding each of the scores. Eachof the plurality of scores is compared with a first threshold value, andthe number of scores above (or below) the first threshold value may becounted, representing the number of tiles of the image with scores above(or below) the first threshold. Then, this number of scores may becompared with a second threshold value, and the single predicted outcomevalue may be a binary value (representing good/bad) depending on whetherthe number of scores is above or below the second threshold value.Accordingly, the method may further include determining a firstthreshold value and a second threshold value, by for the plurality ofmicroscopic images, comparing each of the outcome scores of theplurality of tiles with a first threshold value, counting the number ofscores above or below the first threshold value; and comparing thecounted number of scores with a second threshold value to obtain abinary single predicted outcome value depending on whether the countednumber of scores is above or below the second threshold; and optimisingthe first and second threshold values to minimise the variance betweenthe binary single predicted outcome value and the respective measure ofoutcome for the plurality of images. The resulting computer implementedtrained machine learning algorithm provides a useful means for analysinghistopathological images and providing useful outputs which can be usedto guide medical decision-making. Indeed, it is possible to supply adata carrier or computer with the trained machine learning algorithmimplemented upon it to medical facilities with limited numbers oftrained staff to provide high quality histopathological analysis to suchmedical facilities.

According to a second aspect of the invention, there is provided acomputer-implemented analysis method for histopathological imageanalysis, using a machine-learning algorithm trained according to amethod as summarised above, the analysis method comprising:

obtaining, by the computer, a test microscopic image of a histologicalspecimen that has been stained with the marker;

providing, by the computer, the trained machine-learning algorithm;dividing the first microscopic image into a plurality of tiles;

evaluating the plurality of tiles using the trained machine-learningalgorithm to deliver a plurality of scores corresponding to theplurality of tiles; and

outputting a representation of the plurality of scores for the testmicroscopic image.

The trained machine-learning algorithm can perform surprisingly betterthan a human expert at this task.

In one embodiment, the step of outputting a representation of theplurality of scores includes outputting an image of the tiles and avisual representation of each of the scores. In a particular embodiment,the visual representation of each tile is a grey scale or colourcorresponding to the score to generate a “heat map” of the testmicroscopic image, i.e. an image corresponding to the test microscopicimage but with each tile represented by a grey-scale or colour valuecorresponding to the score. For example, red may be used for highscores, blue for low scores, and orange, yellow and green used forintermediate scores. This allows the visual representation to indicateto the viewer where in the original test image potentially dangeroustissue may be present. The viewer may then examine these parts of theoriginal image more closely.

The measure of outcome used in the training algorithm is typically abinary value (for example representing good and bad outcomes) oralternatively has only a few potential values (for examplegood/average/bad). Surprisingly, even with such a limited range ofoutcomes each applied to not just a single tile but to all tiles of animage, it is still possible to generate as the output of thecomputer-implemented method for histopathological image analysis a “heatmap” showing where the tissue may show pathological features. In otherwords, the trained machine-learning outputs can generate usefulinformation that goes beyond the simple good/bad (or similar) outcomeinformation used in the training inputs.

In another embodiment, the step of outputting a representation of theplurality of scores includes outputting a single outcome value obtainedby combining the plurality of scores to achieve a single predictedoutcome value of the test microscopic image. This single predictedoutcome value represents the whole of the test microscopic image andprovides a simple representation of the image.

Note that in some embodiments the score output by the machine-learningalgorithm for the respective tiles is a single number which can take anyof a range of values, while in many practical applications the measureof outcome may typically be a simple binary value (good/bad). During thetraining process of the machine-learning outcome, the same firstthreshold value may be used to turn the score representing the tile intoa simple binary value (good/bad) for comparing with the respectivemeasures of outcome. In this case, the score output by themachine-learning algorithm provides additional information which may beconsidered to represent how “sure” the machine-20 learning algorithm isabout the result.

In a particularly preferred embodiment, the step of outputting arepresentation of the plurality of scores includes outputting both avisual representation of each of the scores and the single predictedoutcome value. In embodiments, the machine learning algorithm is aneural network. In a particular embodiment, a convolutional neuralnetwork is used.

In a preferred embodiment, the method includes obtaining the pluralityof microscopic images of histological specimens that have been stainedwith a marker using at least two different pieces of image scanningequipment. By avoiding the use of a single piece of image scanningequipment, the trained machine-learning algorithm is not trained to useonly the images from that single piece of image scanning equipment, andaccordingly the trained machine-learning algorithm should be morecapable of processing images from different pieces of image scanningequipment.

In the case that different pieces of image scanning equipment are used,the method may further include a step of aligning the images. This maybe done by a scale-invariant feature transform, generally referred to asa SIFT transform, for example.

Each microscopic image is preferably a grayscale or colour imageconsisting of one, two or three colour channels. Most preferably, it isa colour image consisting of three colour channels. Thus, it providesthree samples for each pixel. The samples are coordinates in athree-dimensional colour space. Suitable 3-D colour spaces include butare not limited to RGB, HSV, YCbCr, and YUV.

The method may further comprise obtaining, by the computer, a region ofinterest in each of the first microscopic images, the method comprisingtraining the machine-learning algorithm with said regions of interestand excluding from the training any regions other than said regions ofinterest.

In some embodiments, the region of interest may be generated by thecomputer for example, using an image segmentation algorithm. In someembodiments, the region of interest may be input by a user, such as ahistopathologist. In some embodiments, the region of interest may begenerated semi-automatically, with some user input combined with someimage segmentation by the computer. The first marker may comprisehaematoxylin and eosin.

Haematoxylin and Eosin (H&E) dye is relatively cheap and widely used inthe art to stain histological specimens. It is particularly advantageousto be able to perform, using images of H&E-stained specimens,histopathological analysis that could previously only be performed withother, more expensive, or more time consuming marking techniques. Suchanalysis can be performed according to embodiments of the presentinvention. In one embodiment, the step of outputting a representation ofthe plurality of scores includes outputting an image of the tiles and avisual representation of each of the scores. In a particular embodiment,the visual representation includes a grey scale or colour correspondingto the score to generate a heat map of the test microscopic image.

In another embodiment, the step of outputting a representation of theplurality of scores includes outputting an outcome score in the form ofa single data point representing the test microscopic image, the singledata point being obtained by combining the plurality of scores toachieve an overall score for the test microscopic image. In oneembodiment, the single data point outcome score may be achieved bythresholding each of the scores output by the machine learning algorithmfor the tiles of the image, using the first and the second thresholds asdescribed above.

In an alternative embodiment, the single data point outcome score mayuse the first threshold only, by outputting the number of tiles of thetest image which exceed (or which do not exceed) the first threshold.

The method may further comprise conducting, by the computer, automatedanalysis of the heat map and the single output data point. Alsodisclosed is a computer program product comprising a non-transitorycomputer-readable medium having embodied thereon a computer programcomprising computer program code configured to control a computer toexecute all the steps of a method as summarised above when said computerprogram is run on the computer. Also disclosed is a histopathologicalimage analysis apparatus comprising: a computer-readable storage medium;

a memory;

one or more interfaces; and

a processor configured to perform a method as summarised above.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described by way of example with reference tothe accompanying drawings, in which:

FIG. 1 is a block diagram schematically illustrating an exemplarycomputer system upon which embodiments of the present invention may run;

FIG. 2 is a flowchart of a method of training a machine-learningalgorithm, according to an embodiment of a first aspect of theinvention;

FIG. 3 is a flowchart of a method of optimisation used in the method ofFIG. 2;

FIG. 4 is a flowchart showing a method for histopathological imageanalysis, according to an embodiment of a second aspect of theinvention;

FIG. 5 is a chart of patient outcomes for training data;

FIG. 6 is a chart of patient outcomes for input data;

FIG. 7 is a photomicrograph corresponding to a good outcome;

FIG. 8 is a photomicrograph corresponding to a poor outcome.

DETAILED DESCRIPTION

FIG. 1 of the accompanying drawings schematically illustrates anexemplary computer system 100 upon which embodiments of the presentinvention may run. The exemplary computer system 100 comprises acomputer-readable storage medium 102, a memory 104, a processor 106 andone or more interfaces 108, which are all linked together over one ormore communication busses 110. The exemplary computer system 100 maytake the form of a conventional computer system, such as, for example, adesktop computer, a personal computer, a laptop, a tablet, a smartphone, a smart watch, a virtual reality headset, a server, a mainframecomputer, and so on. In some embodiments, it may be embedded in amicroscopy apparatus, such as a virtual slide microscope capable ofwhole slide imaging.

The computer-readable storage medium 102 and/or the memory 104 may storeone or more computer programs (or software or code) and/or data. Thecomputer programs stored in the computer-readable storage medium 102 mayinclude an operating system for the processor 106 to execute in orderfor the computer system 100 to function. The computer programs stored inthe computer-readable storage medium 102 and/or the memory 104 mayinclude computer programs according to embodiments of the invention orcomputer programs that, when executed by the processor 106, cause theprocessor 106 to carry out a method according to an embodiment of theinvention

The processor 106 may be any data processing unit suitable for executingone or more computer readable program instructions, such as thosebelonging to computer programs stored in the computer-readable storagemedium 102 and/or the memory 104. As part of the execution of one ormore computer-readable program instructions, the processor 106 may storedata to and/or read data from the computer-readable storage medium 102and/or the memory 104. The processor 106 may comprise a single dataprocessing unit or multiple data processing units operating in parallelor in cooperation with each other. In a particularly preferredembodiment, the processor 106 may comprise one or more GraphicsProcessing Units (GPUs). GPUs are well suited to the kinds ofcalculations involved in training and using machine-learning algorithmssuch as those disclosed herein. The processor 106 may, as part of theexecution of one or more computer readable program instructions, storedata to and/or read data from the computer-readable storage medium 102and/or the memory 104.

The one or more interfaces 108 may comprise a network interface enablingthe computer system 100 to communicate with other computer systemsacross a network. The network may be any kind of network suitable fortransmitting or communicating data from one computer system to another.For example, the network could comprise one or more of a local areanetwork, a wide area network, a metropolitan area network, the internet,a wireless communications network, and so on. The computer system 100may communicate with other computer systems over the network via anysuitable communication mechanism/protocol. The processor 106 maycommunicate with the network interface via the one or more communicationbusses 110 to cause the network interface to send data and/or commandsto another computer system over the network. Similarly, the one or morecommunication busses 110 enable the processor 106 to operate on dataand/or commands received by the computer system 100 via the networkinterface from other computer systems over the network.

The interface 108 may alternatively or additionally comprise a userinput interface and/or a user output interface. The user input interfacemay be arranged to receive input from a user, or operator, of the system100. The user may provide this input via one or more user input devices(not shown), such as a mouse (or other pointing device, track-ball orkeyboard. The user output interface may be arranged to provide agraphical/visual output to a user or operator of the system 100 on adisplay (or monitor or screen) (not shown). The processor 106 mayinstruct the user output interface to form an image/video signal whichcauses the display to show a desired graphical output. The display maybe touch-sensitive enabling the user to provide an input by touching orpressing the display.

According to embodiments of the invention, the interface 108 mayalternatively or additionally comprise an interface to a digitalmicroscope or other microscopy system. For example, the interface 108may comprise an interface to a virtual microscopy apparatus capable ofWhole Slide Imaging (WSI). In WSI, a virtual slide is generated byhigh-resolution scanning of a glass slide by a slide scanner. Thescanning is typically done piecewise and the resulting images arestitched together to form one very large image at the highestmagnification of which the scanner is capable. These images may havedimensions of the order of 100,000×200,000 pixels—in other words, theymay contain several billion pixels. According to some embodiments, thecomputer system 100 may control the microscopy apparatus through theinterface 108 to scan slides containing specimens. The computer system100 may thus obtain microscopic images of histological specimens fromthe microscopy apparatus, received through the interface 108. It will beappreciated that the architecture of the computer system 100 illustratedin FIG. 1 and described above is merely exemplary and that systemshaving different architectures using alternative components or usingmore components (or fewer) may be used instead.

FIG. 2 is a flowchart showing a method of training a machine-learningalgorithm to perform histopathological image analysis, according to anembodiment of a first aspect of the invention. The method uses a set oftraining images to train the algorithm to detect a histological featureof interest. In particular, it trains the algorithm to detect, in imagesstained with one marker, a histological feature that is usually detected(and typically is more easily detectable by a human expert) in imagesstained with another, different marker.

In step 210, the computer 100 obtains a plurality of first colourmicroscopic images of microscopic images of histological specimens thathave been stained with a marker.

The colour images may be obtained by controlling a Virtual Microscope(VM) to scan slides containing the specimens. One suitable group ofvirtual microscopes is sold by Hamamatsu Photonics of Japan, under theproduct name “NanoZoomer”. The virtual microscope comprises microscopeoptical components, a stage for mounting the specimen to be examined, aCharge-Coupled Device (CCD) array or other electronic imaging device forreceiving the image of the specimen, a computer for processing the imageand a Visual Display Unit (VDU) for displaying the image and other data.A prepared slide containing a slice of biological tissue is scanned bythe virtual microscope to produce a respective first colour microscopicimage. For each point (pixel) on the image, the CCD array includes red,green, and blue wavelength detectors, providing respective red, green,and blue signals. Thus, the virtual microscope produces a colour imageconsisting of three colour channels. In the present example, the maximummagnification provided by the virtual microscope corresponds toapproximately a 40× objective optical microscope lens when used on astandard bright field microscope.

Other ways of obtaining the images are also possible. Indeed, in thetraining phase, these microscopic slides from which the images areobtained should the outcomes for the individual patients are known. Theimages are then paired up with a value indicating an outcome such ascancer-specific survival time and provided to the computer 100.

In a preferred embodiment, the images are obtained on a variety ofdifferent pieces of scanning equipment. This is so that in thesubsequent training phase the machine learning algorithm is not trainedon features specific to a particular piece of scanning equipment. Inthis way, the trained machine learning algorithm may have more generalapplicability The annotation as drawn on one scan can be transferred toanother by the use of a scale-invariant feature transform, generallyreferred to as a SIFT transform, for example.

Examples of the images captured, in black and white, are presented inFIGS. 7 and 8. FIG. 7 is an example of an image corresponding to a goodoutcome and FIG. 8 is an example of an image corresponding to a pooroutcome.

In step 220, the images are divided into tiles, typically at least 250and in embodiments 500 or 1 000 tiles or more. Each tile is assigned theoutcome of the individual image. In step 230, a machine learningalgorithm is trained on the tiles (not the whole image) and theoutcomes. The large number of tiles compared with images creates asignificant dataset to be used to train the algorithm.

In the specific embodiment, the machine learning algorithm is a neuralnetwork, in particular a convolutional neural network. However, othermachine learning algorithms may be used, for example other designs ofneural network.

This step generates a trained neural network algorithm 245. In onearrangement, the outcome score output by the trained neural networkalgorithm 245 is a binary number representing a good/bad outcome whichcan be directly compared with a binary good/bad However, in a preferredembodiment, the outcome score of the trained neural network for eachtile is a number that can take more than two values, i.e. a number on ascale representing good or bad the outcome based on the image content ofthe tile.

However, a further step is taken during this training phase in thisspecific embodiment. The outcome score of the trained neural networkalgorithm 245 is an output for each tile. However, in a subsequent stepof using the algorithm 245 to evaluate an image, there is a need for anoutput for each image, not just each tile, as will be described later.In order for this output to be generated, it is necessary to obtainthresholds, as will now be explained.

In order to provide a single outcome score for each image in step 240 afirst threshold and a second threshold are obtained. This is done by anoptimization process, automatically adjusting the first and secondthresholds to obtain the best results. The optimisation process isillustrated in FIG. 3.

In essence, the optimisation works by calculating a single output foreach of the input images. In a first step 242, for each of the tiles ofeach the microscopic images used for training the output of the trainedneural network algorithm 245 is compared with a first threshold, givingan essentially binary value for each tile (good/bad). In a second step244, for each image, the number of tiles having one of the binary valuesis compared with a second threshold, giving an essentially binary valuefor the image as a whole.

This process is repeated (step 246) varying the first and secondthresholds until the best match to the outcomes input in step 210 isobtained, and the first and second thresholds are then output (step248).

Such optimization may use Nelder-Mead, quasi-Newton or conjugategradient algorithms. The function to be optimized is the differencebetween the trained machine learning output for each training image andthe single predicted outcome value calculated by the trainedmachine-learning algorithm for each respective image. Since thisoptimization only requires the selection of two variables (the first andsecond threshold values) and the calculations are only required for arelatively small number of training images, in contrast to therelatively large number of tiles, there is no need for the use of neuralnetwork techniques, though they may be used as an alternative ifdesired.

A conventional optimisation approach can be used. Such conventionaloptimisation algorithms are known in the art and will not be describedfurther. For example, the optimisation algorithm described athttps://stat.ethz.ch/R-manual/R-devel/library/stats/html/optim.html maybe used.

Alternatively, the determination of the first and second thresholds maybe carried out at a later stage, not during the same process asgenerating the trained neural network machine-learning algorithm.

The use of the trained database will now be discussed with reference toFIG. 4.

Firstly, a histological microscopic test image is obtained (step 250)and input into computer 100. The area of the tumour is delineated. Inthe specific examples presented below, this step was carried out by apathologist. However, in alternative embodiments, this step may also beautomated. Then, the test image is divided into a plurality of tiles(step 260).

For each of the tiles, the trained neural network 245 is used to providethe outcome score for each tile, in the form of a number on a scale(Step 270).

The outputs are then generated. In this embodiment, there are twooutputs, a heat map and a single overall score.

To generate the heat map, i.e. a digital image, a representation of theoutcome of each tile is used. An image is created of the tiles, arrangedas they are in the individual image, and for each tile the colour of thetile in the heat map represents the outcome score. For example, a highscore may be represented by a white or yellow colour, and a low score bya blue or green colour.

In alternative embodiments, a grey scale may be used in which highscores are represented by white, low scores by black, and intermediatescores by shades of grey, or alternative mappings may also be used.

Such a heat map gives the user an indication of where in the imagepotentially relevant pathological tissue may be present.

To generate the overall score, the outcome score for each of the tilesof the test image is compared with the first threshold obtained in step240, giving an essentially binary value for each tile (good/bad). Thenthe number of tiles having one of the binary values is compared with thesecond threshold, giving an essentially binary value for the image as awhole, which is output together with the heat map.

In alternative embodiments, only one of these outputs or alternativeoutputs may be provided as long as they represent the outcomes of thetiles of the image.

This approach was trialled with a dataset for which patient outcomes arealready known, and the results presented with reference to FIGS. 5 and6.

In a first embodiment, the above method and apparatus was used to traina convolutional neural network with 169 training microscopic images,corresponding to 169 patients of which 83 had good outcomes and 86 pooroutcomes. Note that the images created from specimens obtained fromcases where outcomes for the individual patients are accordingly known.The images were colour representations (RGB) of H and E stains of tissuesections from the colons of the 169 patients.

After the training, the trained neural network was used to predictoutcomes for the same 169 images. The predicted outcome was comparedwith cancer-specific survival for a number of years as illustrated inFIG. 5. As expected, when presented with the same patients and sameimages on which the neural network was trained, the patients for which agood outcome was predicted had much better outcomes than the patientsfor which a poor outcome is predicted.

Of course, this result is expected and does not demonstrate the utilityof the trained neural network for other data.

Accordingly, the same trained neural network was used to evaluate afurther 76 patients that had not been not used in the training. Thepredicted outcome (good or poor) was predicted for each patient,resulting in a prediction of a good outcome for 36 patients and a pooroutcome for 40 patients. This was then compared with the cancer-specificsurvival as illustrated in FIG. 6. Even with these different patients,the predictions clearly have considerable merit and the long termoutcome for patients predicted to have poor outcomes is considerablyworse (less than 40% survived 5 years) than for patients predicted tohave good outcomes (80% survived 5 years).

This demonstrates that the trained neural network had useful predictivevalue.

Note that unlike prior art approaches in which the relevant features ofthe images were obtained by humans, in the present case all theinformation that was used to train the neural network was thethree-colour images of the stained section. The only human interventionin the example above was to select the area of the tumour.Alternatively, this step too can be automated. Thus, this approach makesit easier to obtain useful test analysis as there is no need for askilled user to identify relevant features of images. While theinvention has been illustrated and described in detail in the drawingsand foregoing description, such illustration and description are to beconsidered illustrative or exemplary and not restrictive; the inventionis not limited to the disclosed embodiments.

In some embodiments, the method of training the machine-learningalgorithm may be implemented in an iterative fashion. The computer 100may obtain a single first colour microscopic image and a singlecorresponding second colour microscopic image at a time. The computermay process this pair of images and may then check whether a sufficientnumber of images has been obtained to train the machine-learningalgorithm. If not, the computer may repeat the process for a furtherpair of images. This procedure is repeated until the machine-learningalgorithm is fully trained, at which point the iterations stop.

Of course, those skilled in the art will appreciate that controlling avirtual microscope to scan slides is not the only way for the computer100 to obtain colour microscopic images. Libraries of suitable imagesmay already be available from other sources. However, it is noted thattraining methods according to embodiments of the present inventionrequire the existence of two corresponding sets of slide-images—one setin which the specimens are stained with a first marker and a second setin which the specimens are stained with a second marker.

Correspondingly, in the examples described above, the microscopic imageswere colour microscopic images consisting of three colour channels. Inother embodiments, such as those using fluorescent dyes, either or bothof the microscopic images may have fewer than three colour channels. Inparticular, the microscopic images may be grayscale images, consistingof a single channel, or they may be two-channel images.

Although the steps of the training method were described above in aparticular order with reference to FIG. 2, in general the steps need notbe carried out in this order.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. The mere fact that certain measures are recited inmutually different dependent claims does not indicate that a combinationof these measured cannot be used to advantage. Any reference signs inthe claims should not be construed as limiting the scope.

1. A computer-implemented analysis method for histological imageanalysis, the method comprising: obtaining, by the computer, a testhistological image that has been stained with a marker; dividing thetest histological image into a plurality of tiles; evaluating theplurality of tiles using a trained machine-learning algorithm to delivera plurality of scores corresponding to the plurality of tiles; comparinga representation of the plurality of scores with a threshold value; andoutputting a representation of the scores as a single predicted outcomevalue.
 2. The computer-implemented analysis method of claim 1, furthercomprising: counting the number of scores above or below the thresholdvalue; and outputting a representation of the counted number of scoresas a single predicted outcome value.
 3. The computer-implementedanalysis method of claim 2, further comprising: comparing the countednumber of scores with a second threshold value to obtain a binary singlepredicted outcome value depending on whether the counted number ofscores is above or below the second threshold; and outputting arepresentation of the binary single predicted outcome value.
 4. Thecomputer-implemented analysis method of claim 1, further comprising:outputting an image of the tiles and a visual representation of each ofthe scores.
 5. The computer-implemented analysis method of claim 1,wherein the method is for histopathological image analysis.
 6. Thecomputer-implemented analysis method of claim 1, wherein the markercomprises Haematoxylin and Eosin.
 7. The computer-implemented analysismethod of claim 1, wherein the single predicted outcome value representsa prognostic outcome for cancer patients.
 8. The computer-implementedanalysis method of claim 1, wherein the single predicted outcome valuerepresents risk of cancer-specific death.
 9. A computer-implementedmethod of training a machine-learning algorithm to perform histologicalimage analysis, comprising: obtaining, by the computer, a plurality ofhistological images and a measure of outcome for each of the images;dividing each of the plurality of histological images into a pluralityof tiles; and training a machine-learning algorithm to generate anoutcome score for an image tile using as training inputs all of theplurality of tiles of the histological images; determining a thresholdvalue by: for the plurality of histological images, comparing each ofthe outcome scores of the plurality of tiles with the threshold value,counting the number of scores above or below the first threshold valueto obtain a single predicted outcome value; and optimising the thresholdvalue to minimise the variance between the single predicted outcomevalue and the respective measure of outcome for the plurality ofhistological images.
 10. The computer-implemented method of claim 9,further comprising: determining the threshold value and a secondthreshold value, by: counting the number of scores above or below thethreshold value, and comparing the counted number of scores with thesecond threshold value to obtain a binary single predicted outcome valuedepending on whether the counted number of scores is above or below thesecond threshold; and optimising the first and second threshold valuesto minimise the variance between the binary single predicted outcomevalue and the respective measure of outcome for the plurality ofhistological images.
 11. The computer-implemented method of claim 9,further comprising: obtaining the plurality of histological images usingat least two different pieces of image scanning equipment.
 12. Thecomputer-implemented method of claim 9, wherein the plurality ofhistological images have been stained with a marker.
 13. Thecomputer-implemented method of claim 12, wherein the marker comprisesHaematoxylin and Eosin.
 14. The computer-implemented method of claim 9,further comprising: aligning the plurality of histological images usingat least two different pieces of image scanning equipment.
 15. Thecomputer-implemented method of claim 9, wherein the machine-learningalgorithm is a neural network.
 16. The computer-implemented method of15, wherein the neural network is a convolutional neural network. 17.The computer-implemented method of claim 9, wherein each of theplurality of histological images is divided into at least 200 tiles. 18.The computer-implemented method of claim 9, wherein the histologicalimage analysis is histopathological image analysis.
 19. Thecomputer-implemented method of claim 9, wherein the outcome score andthe measure of outcome represent a prognostic outcome for cancerpatients.
 20. The computer-implemented method of claim 9, wherein theoutcome score and the measure of outcome represent risk ofcancer-specific death.